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- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 

- Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1 )g] Responsive to communication^) filed on 20 November 2003 , 
2a)S This action is FINAL. 2b)CD This action is non-final. 

3) CD Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayte, 1935 CD. 1 1 , 453 O.G. 213. 

Disposition of Claims 

4) E3 Claim(s) 25-53 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) CD Claim(s) is/are allowed. 

6) ^ Claim(s) 25-53 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) CD Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) CD The specification is objected to by the Examiner. 

1 0)S The drawing(s) filed on ///ao/os is/are: a)|2 accepted or b)CD objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1.85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

1 1 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-1 52. 

Priority under 35 U.S.C. §§119 and 120 

1 2)CD Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 1 1 9(a)-(d) or (0. 
a)D All b)D Some*c>n None of: 

Certified copies of the priority documents have been received. 
Certified copies of the priority documents have been received in Application No. . 



1. D 

2. D 

3-D 



Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 

13) CD Acknowledgment is made of a claim for domestic priority under 35 U.S.C. § 1 19(e) (to a provisional application) 

since a specific reference was included in the first sentence of the specification or in an Application Data Sheet. 
37 CFR 1.78. 

a) □ The translation of the foreign language provisional application has been received. 

14) D Acknowledgment is made of a claim for domestic priority under 35 U.S.C. §§ 120 and/or 121 since a specific 

reference was included in the first sentence of the specification or in an Application Data Sheet. 37 CFR 1.78. 
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DETAILED ACTION 

1. This communication is in response to request for reconsideration filed 11/20/03, claims 25-53 
remain pending. 

2. Quotation of 35 U.S.C. § 103(a) which forms the basis for all obviousness rejections set forth in 
this Office action may be found in previous office action. 

3. Claims 25-53 are rejected under 35 U.S.C. 103(a) as being unpatentable over Sezan et. al. (Sezan) 
MPEG-7 Standardization Activities in view of Bergman et. al. (Bergman) U.S. Patent No. 6,564,263 Bl. 

Regarding claim 25, Sezan teaches features of the invention substantially as claimed, teaching a 
system/method comprising: 

identifying multimedia categories (types) from received multimedia contents (introduction 
section, page 5 1 7); 

extracting multimedia objects to generate multimedia object descriptions from the multimedia 
content for a multimedia (type) (feature extraction, page 518, descriptors section page 518, Fig. 1, 
description generation); 

generating, from the multimedia object descriptions (description, page 5 1 8), an entity relationship 
representation (graph descriptions) for a multimedia type (features hierarchy section, page 519); however 
prior art does not explicitly teach integrating the multimedia object descriptions and entity relation graph 
descriptions to generate a description record to represent content embedded within the multimedia 
content; 

Bergman teaches aggregating, incorporating or combining, i.e. integrating from the multimedia 
object descriptions (col 12/lines 17-26), entity relation graph descriptions (col 3/lines 27-36, 46-51, 59- 
62, Figs. 3-7, col 4/lines 20-30, integrated multimedia descriptions and entity relationships descriptions: 
col 6/lines 57-64, col 8/lines 42-67, entity-relationships, col 9/lines 55-col 10/line 10) to generate an 
object ("description record"); 

wherein a composite object supports embedding of multimedia contents (col 7/lines 2-6, 
embedded hyper-linking, col 15/lines 4-17, spatial relationship of object within/contained, col 1 7/lines 
10-19, object within another object, col 1 7/lines 49-52); additionally teaching 

identifying multimedia type or modalities in multimedia content (col 3/lines 21-36, description 
data type, col 12/lines 23-64, select identified type: col 1 9/lines 41-49); 
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capture (extract) multimedia object features, attributes or modalities from multimedia objects to 
form multimedia object descriptions and entity relation descriptions from the multimedia object (col 
3/lines 37-51, col 6/lines 39-43, extract: col 19/lines 54-58); 

generating entity relationships descriptions and multimedia object descriptions from a multimedia 
category, type or modality (develop: col 14/Iines 44-48, create: col 12/lines 43-50, generate: col 19/lines 



It would have been obvious to one ordinary skilled in the art at the time the invention was made 
to include Bergman's teachings for aggregating, incorporating or combining, i.e. integrating the 
multimedia object descriptions and their respective entity relation graph descriptions to generate a 
description record to represent content embedded within the multimedia content, motivation would be to 
generate entity relation description based on the multimedia object descriptions for multiple multimedia 
content types including a composite multimedia object description that represents content embedded 
within the multimedia content, as taught by Bergman. 

Regarding claim 26, multimedia object pyramid (hierarchy) descriptions for one of the multimedia types 
(Figs. 3-4 & 8, hierarchical model of multimedia object descriptions based on content type; col 8/lines 12- 
16, 20-67, Sezan: feature hierarchy, page 519). 

Regarding claim 27, the multimedia types includes image (Sezan: media type, page 519). 

Regarding claim 28, separating (segmenting) each multimedia content into descriptor defining portions 
(segments) including content from one of the multimedia type for the multimedia content (Sezan; page 
518); and 

generating one feature representing a feature for one of the portions by feature extraction and 
annotation (Bergman: feature description: col 8/lines 42-46, annotation: col 8/lines 55-58, extraction col 
3/lines 37-51); 

wherein the generated multimedia object descriptions comprises one feature description for one 
segment (Bergman; col 8/lines 20-41). 



50-53); 



Regarding claim 29, wherein the segments are selected from the group consisting of local segments and 
global segments (Bergman: col 8/lines 55-58). 
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Regarding claim 30, feature description from the group consisting of media, semantic and temporal 
features (Bergman: col 8/lines 42-67). 

Regarding claim 31, a feature description selected from the group consisting of data location, scalable 
representation and modality transcoding (Bergman: col 9/lines 25-37). 

Regarding claim 32, wherein the semantic features are further defined by one feature description selected 
from the group consisting who (Bergman: col 8/lines 47-50). 

Regarding claim 33, temporal features are further defined by one feature description consisting of 
duration (Bergman: objects duration, col 16/lines 16-34). 

Regarding claim 34, 

generating media object descriptions from the multimedia segment for one of the multimedia 
types by media object extraction processing (Sezan: feature extraction, page 518, descriptors section page 
518, Fig. 1, description generation, Bergman: capture (extract) multimedia object features, col 3/lines 37- 
51); 

generating media object hierarchy descriptions from the generated media object descriptions by 
object hierarchy construction and extraction processing (Bergman: col 14/lines 44-48; Sezan: description, 
page 518, features hierarchy section, page 5 19); and 

generating media entity relation graph descriptions from the generated media object descriptions 
by entity relation graph generation processing (Bergman: col 3/lines 27-36, 46-51, 59-62, Figs. 3-7, col 
4/Unes 20-30, col 6/lines 57-64, col 8/lines 42-67, entity-relationships, col 9/lines 55-col 10/line 10). 

Regarding claim 35, segmenting the content of each multimedia type in the multimedia object into 
segments within the multimedia object by media segmentation processing (Bergman: col 3/lines 21-36, 
col 12/lines 23-64; Sezan: categories (types) introduction section, page 517); 

generating one feature description for one of the segments by feature extraction and annotation 
(Bergman: feature description: col 8/lines 42-46, annotation: col 8/lines 55-58, extraction col 3/lines 37- 
51); 

wherein the generated media object descriptions comprise the feature description for one of the 
segments (Bergman; col 8/lines 20-41). 



Application/Conl 
Art Unit: 2142 




iber. 09/495,175 




Page 4 



Regarding claim 36, substantially the same as claim 30, same rationale of rejection is applicable. 

Regarding claims 37-40, wherein generating media object pyramid (hierarchy) descriptions generates 
terminal/composite objects define as multimedia content pyramid (hierarchy) descriptions of the media 
object descriptions (Bergman: col 3/lines 27-36) based on relationships of media objects represented by 
the media object descriptions (Bergman: col 3/lines 41-51), and wherein the relationships consisting of 
media feature relationships, semantic feature relationships, temporal feature relationships, and spatial 
feature relationships (Bergman: col 6/line 1-14, 57-67, col 7/lines 10-14, 20-25). 

Regarding claims 4 1 and 46, wherein generating media entity relation graph descriptions (Bergman: col 
9/lines 55-col 10/line 10) generates entity relations graph descriptions of the media object descriptions 
based on entity-relationships and dependency-entity relationships of media objects represented by the 
media object descriptions (Bergman: col 14/lines 45-48, col 19/lines 50-53), 

wherein the relationships are selected from the group consisting of media feature relationships, 
semantic feature relationships, temporal feature relationships and spatial feature relationships (Bergman: 
col 6/lines 1-14, 57-67, col 7/lines 10-14, 20-25). 

Regarding claims 42-45, wherein generating multimedia object pyramid (hierarchy) descriptions 
generates multimedia object hierarchy descriptions of the multimedia object descriptions based on media 
content relationships of multimedia terminal/composite objects represented by the multimedia object 
descriptions (Bergman: col 11/line 27-33, inter-feature relationships: col 19/lines 50-53, inter-object 
relationships, col 15/lines 10-15); based on temporal, spatial feature relationships of multimedia objects 
(Bergman: col 15/lines 10-15); and semantic feature relationships of multimedia object (Bergman: col 
6/lines 58-64). 

Regarding claim 47, receiving (Sezan: page 517) and transcoding (encoding) the multimedia object 
descriptions into encoded description information (Bergman: col 7/lines 20-24), and storing the encoded 
description information as one (description) record (Sezan: left column, page 518). 

Regarding claim 48, this claim combines limitation(s) substantially the same as claims 25, and 47, same 
rationale of rejection is applicable 



Regarding claim 49 & 50, the encoding comprises binary encoding (Bergman: col 1 3/lines 31-33). 
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Regarding claims 51 & 52, the encoding comprises the extensible Markup Language (XML) encoding 
(Bergman: col 14/lines 4-18). 

Regarding claim 53, includes limitations substantially the same as claim 25, same rationale of rejection is 
applicable, and further 

generating, from the multimedia object descriptions, multimedia object pyramid (hierarchy) 
descriptions (Bergman: generating entity relationships descriptions and multimedia object descriptions 
from a multimedia category, type or modality develop: col 14/lines 44-48, create: col 12/lines 43-50, 
generate: col 19/lines 50-53) by object hierarchy construction and extraction processing, for a 
multimedia content modality or feature (type) (extract: col 3/lines 41-51, col 6/lines 39-42, construct: col 
12/lines 43-50); and 

integrating the multimedia object descriptions and the multimedia object pyramid (hierarchy) 
descriptions to generate a descriptor (description record) to represent content embedded within the 
multimedia content (Bergman: integration, col 12/lines 17-26, entity relation graph descriptions, col 3/lines 
27-36, 46-51, 59-62, Figs. 3-7, col 4/lines 20-30, integrated multimedia descriptions and entity 
relationships descriptions: col 6/lines 57-64, col 8/lines 42-67, entity-relationships, col 9/lines 55-coI 
10/line 10) to generate a composite object); 

wherein a composite object supports embedding of multimedia contents (col 7/lines 2-6, 
embedded hyper-linking, col 15/lines 4-17, spatial relationship of object within/contained, col 17/lines 
10-19, object within another object, col 17/lines 49-52). 

Response to Arguments 

4. Applicant in regards to claim 1 argues prior art does not teach claim limitation as recited, that is, 
the generating and integrating limitation of the claim, because prior art's InfoPyramid is not an entity 
relation graph description and prior art composite object cannot be equated to claimed description record. 

It response to the above-mentioned argument, applicant's interpretation of the prior art is noted. 
However, according to applicant's specification, broadly speaking object and events are used as entities of 
description (see page 4/lines 16-20), a description is an instantiation of data, i.e. an object (see page 7, 
lines 16-page 8, line 1), organization of multimedia object using the relationship among them (page 13, 
lines 1-8) by modeling general relationships among multimedia object (see page 14, lines 7-13), spatial 
relationships among single-media objects inside a multimedia object are described using the entity 
relation graph (page 18, lines 12-25), relationships that cannot be expressed using a hierarchical structure 
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(e.g. one object talks to another) are expressed using an entity relation graph, e.g. spatial and temporal 
(i.e. topological or directional) (page 20, line 19-page 23, line 8). Fig. 5 illustrates an entity relation graph. 
Therefore, an entity relation graph description is a graphical representation of objects describing 
relationships among them (e.g. temporal/spatial among others) (as noted on claim 35). 

Prior art Bergman, teaches a description record to represent content embedded within multimedia 
content (col 7/lines 2-6, embedded hyper-linking, col 15/lines 4-17, spatial relationship of object 
within/contained, col 17/lines 10-19, object within another object, col 17/lines 49-52); Further teaching 
where non-terminal objects include objects with spatial, temporal relationships and thus allows the 
description of these relationships between presentation object (col 3/lines 37-51, see Fig. 5-6), the 
composite object (non-terminal) use graphic block-based representation to describe spatial and temporal 
relationships between the objects (col 5/lines 64-col 6/line 14). The Info Pyramid is a representation 
model of description scheme for describing object the representation includes individual modalities as 
well as additional modalities that include spatial and temporal characteristics, the multimedia content 
description framework or scheme provides a way to represent both spatial and temporal relationships 
among multiple object as well as inter-object user interactions (col 6/lines 39-67), Figs. 5-6 illustrated an 
inter object specification of object with spatial and temporal relationship (i.e. entity relation graph 
description). Thus, prior art teaches a "description record" and an "entity relation graph" as claimed and 
within the scope of applicant's invention. 

5. Applicant's arguments filed 1 1/20/03 have been fully considered but not rendered persuasive. 

6. This is a request for continued examination (RCE) filed under 37 CFR 1. 1 14 of applicant's earlier 
Application No. 09/495,175. All claims are drawn to the same invention claimed in the earlier 
application and could have been finally rejected on the grounds and art of record in the next Office action 
if they had been entered in the earlier application. Accordingly, THIS ACTION IS MADE FINAL even 
though it is a first action in this case. See MPEP § 706.07(b). Applicant is reminded of the extension of 
time policy as set forth in 37 CFR 1. 136(a). 

A shortened statutory period for reply to this final action is set to expire THREE MONTHS from 
the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing 
date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH 
shortened statutory period, then the shortened statutory period will expire on the date the advisory action 
is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of 
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the advisory action. In no, however, event will the statutory period for reply expire later than SIX 
MONTHS from the mailing date of this final action. 



may request continued examination of the application by filing a Request for Continued Examination of 
under 37 CFR § 1.1 14 and providing the corresponding fee set forth in § 1.17(e) for the submission of, 
but not limited to, new arguments, an information disclosure statement, an amendment to the written 
description, claims, drawings, or new evidence in support of patentability. Or applicant whose claims has 
been twice rejected, may appeal from the decision of the administrative patent judge to the Board of 
Patent Appeals and Interferences under 35 U.S.C. §134. 

Any inquiry concerning this communication or earlier communications from the examiner should 
be directed to Prieto, B. whose telephone number is (703) 305-0750. The Examiner can normally be 
reached on Monday-Friday from 6:00 to 3:30 p.m. If attempts to reach the examiner by telephone are 
unsuccessful, the Examiner's Supervisor, Jack B. Harvey can be reached on (703) 305-9705. Any inquiry 
of a general nature or relating to the status of this application or proceeding should be directed to the 
receptionist whose telephone number is (703) 305-3800/4700. 

Any response to this final action should be mailed to: 

Box AF 

Commissioner of Patents and Trademarks 
Washington, D C. 20231 

or faxed to the Central Fax Office: 

(703) 872-9306, for Official communications and entry 

Or Telephone: 

(703) 306-563 1 for TC 2100 Customer Service Office 

Hand-delivered responses should be brought to Crystal Park II, 2121 Crystal Drive, Arlington 
VA, Sixth Floor (Receptionist). 



Prosecution of this application is closed by means of this final office action § 1.1 13, applicant 



B. Prieto 

TC2100 
Patent Examiner 
January 16, 2004 




